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Abstract 

Different  organizational  structures  are  better  matched  to  certain  mission  types  than  others 
(organizational  congruence).  Consequently,  one  way  to  achieve  superior  mission 
effectiveness  is  to  switch  between  organizational  structures  when  circumstances  dictate. 
However,  little  is  known  about  the  variables  that  signal  the  need  for  such  structural 
adaptation.  To  explore  this  issue,  we  used  a  model-based  design  process  to  create 
mission  scenarios  that  were  either  matched  (congruent)  or  mismatched  (incongruent)  with 
two  organizational  structures  (Functional,  Divisional).  Results  indicated  that,  as 
predicted  on  the  basis  of  the  coordination  requirements  imposed  by  the  model-based 
design  process,  performance  in  the  incongruent  cases  was  characterized  by  increased 
communication,  increased  perceived  workload,  and  degraded  performance.  Given  these 
overall  results,  we  explored  these  data  further  by  analyzing  communication  patterns  to 
identify  how  the  organizations  attempted  to  cope  with  the  congruence  problem.  Our 
results  indicated  that  the  communication  strategies  employed  in  the  face  of  incongruence 
depended  on  organizational  structure/mission  scenario  pairings,  suggesting  that  the 
specific  signals  of  the  need  for  structural  adaptation  will  likely  depend  on  context. 

Introduction 

Mission  effectiveness  depends  on  a  number  of  factors,  ranging  from  training  excellence 
to  technical  superiority  to  effective  planning.  In  the  work  reported  here,  we  focus  on  one 
contributing  factor,  “congruence,”  which  has  until  recent  years  gone  largely  unaddressed. 
From  a  human  engineering  perspective,  congruence  is  about  the  “fit”  between 
organizational  structures,  technology,  people,  and  mission  requirements.  Mission 
effectiveness  can  be  optimized  when  these  items  are  well  matched.  In  particular,  here  we 
focus  on  the  central  theme  of  the  Adaptive  Architectures  for  Command  and  Control 
(A2C2)  Research  Program,  which  is  creation  of  organizations  that  fit  mission 
requirements. 


The  research  reported  here  was  sponsored  by  the  Office  of  Naval  Research,  Contract  No.  N00014-02-C- 
0233,  under  the  direction  of  Gerald  Malecki. 


2 


Broadly  speaking,  different  organizations  may  be  better  matched  to  certain  missions  than 
others,  and  in  fact,  empirical  results  indicate  that  modeling  techniques  can  be  used  to 
design  organizations  such  that  there  is  a  tight  fit  or  alignment  between  their  structures  and 
mission  requirements  (e.g.,  Entin,  1999;  Hocevar,  2000;  Hutchins,  Hocevar,  Kemple, 
Kleinman,  Entin,  and  Serfaty,  2000;  Levchuk,  Merina,  Levchuk,  Pattipati,  &  Kleinman, 
2001).  In  other  words,  organizational  effectiveness  can  be  mediated  by  the  congruence 
between  an  organization’s  structural  design  and  an  environment  or  task  (e.g.,  Donaldson, 
2001;  Van  de  Ven  &  Drazin,  1985).  If  organizational  structure  is  out  of  “alignment”  with 
the  organization’s  mission,  then  quality  of  performance  should  be  reduced.  Performance 
may  be  affected  by  whether  the  right  person  has  the  right  resources  at  the  right  place  at 
the  right  time. 

Accordingly,  one  way  to  achieve  superior  mission  effectiveness  may  be  to  change 
organizational  structures  when  circumstances  dictate  in  order  to  achieve  a  better  fit. 
However,  little  is  known  about  when  and  how  organizations  can  and  should  adapt  their 
structures,  and  about  the  variables  that  signal  the  need  for  such  structural  adaptation.  This 
paper  addresses  this  issue  by  focusing  on  an  experimental  study  of  how  organizations 
cope  with  incongruence.  Based  on  our  modeling  approach,  our  empirical  strategy  was  to 
contrast  performance  under  conditions  in  which  organizational  structures  and  missions 
were  congruent  with  performance  under  conditions  in  which  they  were  incongruent. 

Ultimately,  through  this  work,  our  goal  is  to  identify  “leading  indicators”  that  signal  the 
need  to  adapt  organizational  structure  in  order  to  enhance  mission  effectiveness.  We 
define  leading  indicators  as  measures  that  signal  that  an  organization  is  “out  of 
alignment”  with  its  mission  and  needs  to  change  its  structure  to  improve  performance. 
However,  the  ability  to  accurately  identify  leading  indicators  depends  on  the  creation  of 
conditions  in  which  an  organizational  structure  is  measurably  incongruent  with  its 
mission,  and  in  which  this  incongruence  results  in  significant  performance  decrements. 
Given  the  presence  of  observed  decreases  in  performance  due  to  incongruence,  we  can 
then  ask  what  observable  behaviors  were  present  that  could  be  used  as  signals  of 
misalignment. 

Toward  this  goal,  this  paper  addresses:  1)  The  effectiveness  of  our  model-based 
congruence  manipulation,  as  fully  presented  in  Kleinman,  Levchuk,  Hutchins,  &  Kemple 
(2003)  and  Levchuk,  Kleinman,  Ruan,  &  Pattipati  (2003);  and  2)  The  ways  in  which  the 
organizations  studied  attempted  to  cope  with  incongruence.  In  this  paper,  we  address 
these  items  by  focusing  on  performance  measures  over  the  entire  mission  scenarios.  In 
this  manner,  we  verify  the  basic  manipulation  and  provide  initial  insight  into  how 
incongruence  affected  organizational  performance  and  process.  This  work  therefore 
speaks  to  identification  of  leading  indicators.  Moreover,  this  work  sets  the  stage  for 
exploration  of  performance  and  process  changes  over  time,  as  presented  in  our 
companion  paper,  Entin,  Diedrich,  Kleinman,  Kemple,  Hocevar,  Rubineau,  &  Serfaty 
(2003),  which  directly  addresses  what  incongruence  looks  like  in  action.  This  second 
paper  focuses  on  performance  and  processes  over  time,  for  ultimately,  the  goal  is  to 
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identify  the  state  of  incongruence  early  in  a  mission,  long  before  performance 
significantly  depreciates. 

Engineering  of  Organizational  Congruence 

Our  research  strategy  is  to  use  a  set  of  modeling  techniques  to  design  organizational 
structures  and  mission  scenarios.  The  modeling  techniques,  based  on  a  variety  of 
approaches  (e.g.,  Carley  &  Lee,  1998;  Handley,  Zaidi,  &  Levis,  1999;  Levchuk,  Levchuk, 
Luo,  Pattipati,  &  Kleinman,  2002a,  2002b),  are  used  first  for  organizational  design  and 
then  for  simulation  to  evaluate  proposed  designs  prior  to  implementation  and 
experimentation.  In  the  case  presented  here,  our  basic  approach  to  studying  congruence 
was  to  define  two  disparate  organizational  structures  and  then  design  two  missions 
(scenarios)  that  exploited  the  differences  between  the  two  structures.  Thus,  the  objective 
was  for  the  first  mission  scenario  to  be  “matched”  to  organization  1  through  a  high  degree 
of  congruence,  while  also  being  “mismatched”  (i.e.,  exhibit  low  congruence)  with 
organization  2.  The  goal  was  for  the  reverse  to  be  true  for  the  second  scenario.  The 
organizational  structures  and  scenarios  were  implemented  through  the  use  of  the 
Distributed  Dynamic  Decision-making  (DDD)  simulation  testbed. 

We  begin  by  addressing  the  creation  of  the  organizations  and  mission  scenarios  used  to 
investigate  the  congruence  concept.  In  this  paper,  we  briefly  outline  the  strategy,  whereas 
complete  details  can  be  found  in  Kleinman  et  al.  (2003)  and  Levchuk  et  al.  (2003). 
Based  on  these  designs,  we  then  outline  our  primary  hypotheses  and  experimental 
methods. 

Organizational  Structure 

Building  on  previous  work  (Diedrich,  Hocevar,  Entin,  Hutchins,  Kemple,  &  Kleinman, 
2002;  Hutchins,  Kleinman,  Hocevar,  Kemple,  &  Porter,  2001;  Moon,  Hollenbeck,  Ilgen, 
West,  Ellis,  Humphrey,  &  Porter,  2000),  the  two  organizational  structures  that  we 
explored  are  commonly  referred  to  as  Functional  (F)  and  Divisional  (D).  The  functional 
structure  was  organized  such  that  each  participant  specialized  in  one  or  two  aspects  of  a 
mission  such  as  Strike  or  Air  Warfare,  where  the  specific  assets  controlled  were 
distributed  across  multiple  platforms  (ships).  In  contrast,  in  the  divisional  structure,  each 
participant  had  control  over  a  single  multifunctional  platform  that  was  able  to  process  a 
variety  of  functional  tasks  in  a  given  location. 

These  organizational  structures  are  illustrated  in  Table  1,  which  shows  the  asset 
ownership  for  the  two  structures.  Note  that  within  the  DDD,  organizational  structure  is 
operationally  defined  on  the  basis  of  asset  ownership  and  location.  However,  more 
generally,  this  operational  definition  reflects  differences  in  roles  (single  vs. 
multifunctional  areas)  and  geographic  responsibilities  (local  vs.  global  responsibility  in 
area  of  operations).  For  instance,  in  the  functional  organization  (Table  1,  columns),  each 
of  the  six  decision  makers  (DMs)  had  control  over  one  (or  two)  areas  (Strike,  BMD,  ISR, 
AWC,  SuWC/Mines,  SOF/SAR)  with  assets  assigned  across  all  of  the  geographical  areas 
(e.g.,  F18S,  Anti-ballistic  missiles,  UAVs,  F18A,  etc.).  In  contrast,  in  the  divisional 
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organization  (Table  1,  rows),  each  of  the  six  DMs  controlled  assets  across  almost  all  of 
the  functional  areas,  but  these  assets  were  concentrated  in  proximity  to  their  primary 
platform  (ship).  Figure  1  shows  the  overall  view  of  the  area  of  operations,  where  the 
primary  platforms  are  indicated  by  their  colors  and  specifications.  Note  that  the  ranges 
of  the  various  assets  varied  depending  on  the  qualities  of  the  modeled  systems. 


1 

2 

3 

4 

5 

6 

Platform 

STRIKE 

ISR 

AWC 

SuWC/MINES 

SOF/SAR 

1 

CVN 

2F18S 

XXX 

1UAV 

2F18A,  E2C 

1FAB,  1MH53 

1HH60 

2 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1  HH60,1SOF 

3 

DDGB 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1  HH60,1SOF 

4 

CG 

8TLAM 

3  ABM 

1UAV 

6SM2 

1  FAB,2HARP,1  MH53 

1HH60 

5 

FFG* 

2F18S 

XXX 

1UAV 

2F18A,E2C,4SM2 

1  FAB,2HARP,1  MH53 

1HH60 

6 

DDGC 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1  HH60,1SOF 

Table  1.  Organizational  structures  where  the  functional  organization  (F)  is  indicated  in 
columns  and  the  divisional  organization  (D)  is  indicated  in  rows. 


Figure  1.  Simulation  interface  display  showing  battle  space. 

Design  of  Mission  Scenarios 

Given  these  organizational  structures,  our  goal  was  to  use  the  modeling  process  to  design 
a  functional  scenario  (f)  that  fit  the  functional  organization  (F),  while  being  misfit  to  the 
divisional  organization  (D).  Similarly,  we  wanted  to  design  a  divisional  scenario  (d)  that 
fit  D  but  not  F.  Several  factors  went  into  the  design  of  these  two  scenarios  (see  also, 
Kleinman  et  al.,  2003),  but  there  were  two  primary  strategies:  First,  we  manipulated  the 
selection  of  resource  requirements  needed  to  accomplish  particular  tasks  within  the 
scenarios.  Note  that  the  two  organizations  differed  primarily  with  regard  to  the  assets 
(and  hence  the  resource  capabilities)  that  each  DM  owned.  In  the  models  used  to  design 
the  scenarios,  the  degree  of  predicted  (structural)  congruence  is  inversely  related  to  the 
amount  of  inter-DM  coordination  needed  to  accomplish  the  mission.  Therefore,  by 


5 


adjusting  the  resource  requirements  of  selected  tasks  it  is  possible  to  manipulate  the  inter¬ 
player  coordination  needed  to  successfully  prosecute  selected  tasks  for  a  given 
organization- scenario  pairing.  Accordingly,  we  designed  tasks  within  f  such  that  little 
between-DM  coordination  would  be  needed  within  organization  F,  while  significant 
coordination  would  be  needed  by  organization  D,  with  the  reverse  being  true  for  scenario 
d.  For  example,  a  task  requiring  a  single  asset  in  each  of  three  functional  areas  would 
need  three  players  to  coordinate  in  F,  but  could  be  done  by  a  player  in  D,  provided  that 
he/she  had  assets  in  all  of  the  requisite  areas.  Similarly,  a  task  requiring  multiple  assets  in 
a  single  functional  area  is  well  suited  to  F,  but  would  need  multiple  players  in  D  to 
coordinate  their  assets.  In  general,  this  manipulation  included  all  of  the  key  “mission” 
tasks,  plus  most  of  the  time  critical  tasks. 

Second,  we  manipulated  congruence  through  the  temporal  and  geographical  distribution 
of  tasks.  These  largely  involved  tasks  that  had  only  one  or  two  functional  areas 
represented  in  their  resource  requirements  (air  attackers,  sea  attackers,  etc.),  and  aimed  to 
exploit  the  different  geographical  responsibilities  in  D  versus  F.  Note  that  in  D  players 
had  responsibilities  in  a  defined  area,  while  in  F  players  had  responsibilities  that  covered 
the  entire  battle  space.  Thus,  for  example,  in  scenario  d  we  designed  “waves”  consisting 
of  single  functional  area  attackers  (e.g.,  air)  wherein  the  individual  tasks  within  each 
wave  were  distributed  geographically  over  several  players.  Clearly,  this  would  impose  a 
significant  load  upon  one  DM  (e.g.,  air  warfare)  in  F  in  both  time  and  space,  but  the  load 
would  be  shared  among  several  players  in  D  with  little  or  no  coordination  needed.  In 
scenario  f  the  attack  “waves”  consisted  of  several  different  functional  area  tasks  where 
each  wave  targeted  a  specific  player’s  area  in  D.  Thus,  we  could  overload  a  given  player 
in  D,  but  the  load  would  be  distributed  among  several  players  in  F.  Task  categories  used 
for  these  manipulations  included  items  such  as  hostile  air  and  sea  platforms,  missile 
launchers,  search  and  rescues,  and  mine  clearing  tasks.  In  general,  each  of  the  scenarios 
had  approximately  ten  waves  spaced  uniformly  in  time  over  the  length  of  the  scenario. 

Testing  Organizational  Congruence 

Based  on  these  organizational  and  mission  scenario  designs,  to  test  this  model-based 
method  of  engineering  organizational  congruence,  we  designed  an  experiment  in  which  8 
teams  of  6  participants  each  engaged  in  both  scenarios  in  one  organizational  structure. 
Hence,  4  teams  performed  in  D  and  4  teams  performed  in  F,  and  each  of  these  teams 
played  both  the  d  and  f  scenarios  (order  was  counterbalanced  across  teams).  There  were 
two  matched  (congruent)  cases  (D-d  and  F-f)  and  two  mismatched  (incongruent)  cases 
(D-f  and  F-d). 

Based  on  the  mission  scenarios  and  organizational  definitions,  there  were  three  overall 
predictions: 

•  The  overall  level  of  communications  should  be  higher  in  the  mismatched  cases 
because,  based  on  the  models,  coordination  requirements  were  manipulated  to  be 
much  higher  in  the  mismatched  cases  (i.e.,  players  would  have  to  communicate 
verbally  to  process  tasks  requiring  shared  resources). 
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•  The  overall  level  of  workload  should  be  higher  in  the  mismatch  cases.  Because 
coordination  demands  were  manipulated  to  be  high  in  the  mismatched  cases,  this 
would  result  in  increased  communication  requirements  and  an  increased  need  for 
coordinated  actions  to  prosecute  tasks.  Hence,  we  expected  that  the  perceived 
workload  would  be  higher  in  the  mismatched  cases. 

•  Overall,  performance  should  be  worse  in  the  mismatched  cases  because  the 
greater  coordination  requirements  should  lead  to  more  communications  and  higher 
workloads,  resulting  in  some  decrements  in  the  ability  to  perform  multiple  tasks 
quickly  and  accurately. 

Note  that  these  are  general  predictions  regarding  the  overall  effects  of  the  experimental 
manipulation.  Validation  of  these  predictions  therefore  provides  evidence  that  the 
experimental  manipulations  worked  as  intended,  providing  us  with  a  “license”  to  further 
explore  in  detail  how  the  organizations  studied  attempted  to  cope  with  incongruence,  thus 
shedding  light  on  potential  “leading  indicators”  of  the  need  for  organizational  adaptation. 

Hence,  beyond  these  global  predictions,  we  focus  our  analyses  on  strategies  for  coping 
with  incongruence.  Within  the  context  of  this  investigation,  the  organizations  were  not 
permitted  to  adapt  their  structures,  for  instance,  from  D  to  F  in  the  context  of  the 
functional  scenario.  Nevertheless,  the  organizations  were  free  to  adapt  their  strategies  as 
they  attempted  to  deal  with  the  mismatched  situations.  We  believe  that  these  strategies 
might  offer  insight  into  leading  indicators  of  the  need  for  change  because  they  reflect 
how  an  organization  attempts  to  cope  with  incongruence  prior  to  a  change  in 
organizational  structure.  In  other  words,  it  is  possible  that  strategy  changes  might 
consistently  predict  the  need  for  structural  change,  and  as  such,  the  strategy  changes 
might  serve  as  potential  leading  indicators.  We  therefore  focused  our  analyses  on 
communications,  for  communication  is  the  primary  mechanism  through  which  the 
organizations  could  express  their  attempts  to  deal  collectively  with  the  misfit  situations. 

Moreover,  we  also  focused  our  analyses  on  differences  between  how  the  F  and  D 
organizations  attempted  to  cope  with  the  various  scenarios.  This  is  a  critical  issue,  for  as 
stated  above,  our  ultimate  goal  is  to  understand  the  signals  for  change  (leading 
indicators).  Consequently,  it  is  imperative  to  ascertain  how  organizational  structures  and 
mission  scenarios,  which  may  be  more  or  less  congruent,  interact  to  shape  the  behaviors 
observed.  These  behaviors  (e.g.,  strategies)  will  only  serve  as  valid  leading  indicators  to 
the  extent  that  we  understand  the  contextual  factors  that  shape  performance  and  process. 

Method 


Participants 

Forty-eight  officers  attending  the  Naval  Postgraduate  School  in  Monterey,  CA,  served  as 
participants.  Most  of  the  officers  were  03  or  04  and  several  services  and  nations  were 
represented.  Participants  were  organized  into  eight  teams  of  six  individuals  each. 
Participants  received  no  compensation  for  study  participation. 
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Apparatus 


The  simulation  was  implemented  within  the  Distributed  Dynamic  Decision-making 
(DDD)  environment.  The  DDD  is  a  distributed  client  server  simulation  that  provides  a 
flexible  framework  in  which  to  study  individual  and  team  performance.  In  general,  DDD 
simulations  involve  individual  (and  team)  decision-making  about  complex  situations 
based  on  information  and  resources  provided  by  the  simulation  and  other  team  members 
(Serfaty  &  Kleinman,  1985;  Kleinman  &  Serfaty,  1989).  The  simulation  enables  the 
manipulation  of  variables  such  as  organizational  structure  and  mission  scenario  tasking. 
In  addition,  a  variety  of  performance  measures  can  be  recorded  including  items  such  as 
tasks  processed,  latencies,  and  accuracies.  In  this  case,  the  organizational  structures  were 
implemented  in  the  DDD  simulation  by  varying  the  ownership  of  different  kinds  of 
assets.  Thus,  we  designed  a  functional  organization  (F)  and  a  divisional  organization  (D), 
as  defined  in  Table  1.  With  respect  to  scenario  design,  as  specified  in  the  above  section 
“Engineering  of  Organizational  Congruence,”  we  varied  a  number  of  items  in  order  to 
design  the  functional  (f)  and  divisional  (d)  scenarios  including  task  timings,  locations,  and 
resource  requirements. 

Procedure 

Participants  began  the  experiment  by  signing  an  informed  consent  form  and  then 
completing  two  hours  of  DDD  “buttonology”  training  to  learn  how  to  control  the  various 
assets  and  how  to  use  the  various  functions  contained  in  the  DDD  simulation.  A  second 
two-hour  session  provided  training  designed  to  provide  experience  on  the  skills  necessary 
to  perform  either  the  divisional  or  functional  scenarios,  without  creating  a  bias  for  either 
kind  of  scenario.  In  general,  these  training  scenarios  exposed  participants  to  the  general 
mission  task  flow  and  the  task  resource  requirements.  However,  these  training  scenarios 
were  composites  of  the  actual  mission  scenarios  with  respect  to  resource  requirements, 
and  as  such,  neither  of  the  two  training  scenarios  were  explicitly  designed  to  favor  one 
organizational  structure  over  another.  Following  training,  each  team  engaged  in  a  two- 
hour  data  collection  session  followed  by  a  second  two-hour  data  collection  session.  Thus, 
in  total,  each  team  participated  in  four  hours  of  training  and  four  hours  of  testing  over 
four  sessions  over  multiple  days.  At  the  end  of  the  experiment,  all  teams  were  debriefed 
as  to  the  nature  and  purpose  of  the  experiment. 

Each  team  was  assigned  to  either  the  D  or  F  organizational  structure,  which  was 
maintained  throughout  all  training  and  testing.  Thus,  organizational  structure  was 
manipulated  as  a  between-subjects  variable  with  4  teams  in  F  and  4  teams  in  D.  In 
contrast,  during  each  of  the  two  experimental  sessions,  the  teams  each  performed  the  d 
scenario  and  the  f  scenario.  Thus,  scenario  was  manipulated  as  a  within-subjects  variable. 
Across  the  two  sessions,  the  scenarios  were  counterbalanced  as  d-f-f-d  for  half  the  teams 
and  f-d-d-f  for  half  of  the  teams. 

The  experimental  scenarios  involved  a  Joint  Forces  mission  that  lasted  approximately  35 
minutes  or  until  the  final  Port  task  was  captured,  whichever  came  first.  The  teams  used  a 
variety  of  sea,  land,  and  air  assets  to  complete  a  specified  mission  that  included 
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destroying  or  capturing  a  command  center,  two  naval  bases,  two  air  bases,  and  a  port. 
Along  the  way,  the  teams  encountered  a  variety  of  obstacles  including  hostile  ground,  sea, 
and  air  assets.  In  addition,  players  had  to  defend  neighboring  foreign  friendly  areas  from 
SCUD  missile  attacks,  while  also  defending  friendly  assets.  Critically,  the  mission 
environment  was  significantly  characterized  by  a  variety  of  time-critical  tasks  (e.g., 
SCUD  missile  launchers,  coastal  defense  launchers,  search  and  rescue)  that  required 
immediate  response.  The  area  of  operations  also  contained  neutral  parties  and  hostile 
assets  that  did  not  directly  engage  friendly  forces,  and  therefore  were  deemed  to  be  low 
on  the  list  of  priorities  as  specified  by  the  rules  of  engagement  and  commander’s  intent. 

Although  the  commander’s  intent  and  rules  of  engagement  were  similar  across  the 
organizational  structures,  the  precise  operational  roles  given  to  the  participants  depended 
on  organizational  structure.  Hence,  mission  roles  stressed  multiple  functional 
responsibilities  in  a  given  geographic  region  for  the  divisional  organization,  but 
functionally  specific  responsibilities  across  the  entire  area  of  operations  for  the  functional 
organization.  As  noted  above,  in  general,  in  the  functional  organizational  structure  a 
participant  charged  with  one  aspect  of  the  mission,  such  as  air  warfare,  “owned”  (had 
direct  control  of)  all  of  the  assets  necessary  to  do  that  part  of  the  mission  across  the  area 
of  operations.  In  contrast,  in  the  divisional  organizational  structure,  participants  had 
direct  control  over  all  assets  on  one  multifunctional  platform  (e.g.,  a  carrier  or  a  guided 
missile  cruiser)  that  was  located  in  a  geographically  distinct  region.  However,  there  was 
some  overlap  of  range  such  that  multiple  assets  from  multiple  DMs  could  access  the  same 
targets  in  certain  areas. 

Data  Analysis 

Overall,  the  data  collected  in  the  experiment  included  assessments  of  performance, 
communications,  and  workload.  These  dependent  measures  were  derived  from  three 
different  sources:  Instruments  completed  by  trained  observers,  participant  self-report 
measures,  and  measures  derived  from  the  DDD  simulator.  Below,  we  describe  the 
instruments  employed  in  each  of  these  areas  with  respect  to  global  measures.  Time-series 
based  measures  are  presented  in  Entin  et  al.  (2003). 

Performance  Measures.  In  general,  the  DDD  simulator  enabled  the  measurement  of 
several  variables  related  to  individual  and  team  performance  such  as  latency  to  process  a 
task,  accuracy  in  processing  a  task,  number  of  tasks  process,  the  number  of  task  arrivals 
(tasks  in  the  scenario),  etc.  In  this  paper,  we  focus  on  the  percent  of  tasks  processed  (the 
number  of  tasks  attacked/the  number  of  tasks  arrived  in  the  scenario),  the  accuracy  of  task 
processing  (the  ratio  of  assets  applied  to  assets  required  for  tasks  attacked),  and  the 
latency  of  tasks  attacked  (the  time  from  arrival  in  the  scenario  to  the  time  of  attack). 

Team  Communications.  Using  a  custom  software  tool,  verbal  communications  between 
team  members  were  captured  by  observers  at  an  intermediate  level  of  detail  that 
incorporated  both  semantic  and  quantitative  aspects  of  the  communication  stream.  In 
brief,  during  the  conduct  of  a  scenario  two  observers  listened  to  the  communications  in 
real  time,  and  used  specially  designed  software  and  a  handheld,  touch-sensitive  computer 
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to  code  the  source,  the  recipient,  the  time,  and  the  type  of  the  verbal  communications 
among  the  team  members.  One  of  the  observers  coded  the  communication  for  three  of 
the  team  members,  while  the  other  observer  coded  the  communications  exhibited  by  the 
remaining  three  team  members.  Broadly  speaking,  the  types  of  communications  were 
divided  into  three  basic  categories:  transfers  (e.g.,  provision  of  information),  requests 
(e.g.,  demands  for  information),  and  acknowledgements  (e.g.,  “aye”).  Both  transfers  and 
requests  were,  in  turn,  classified  as  requests/transfers  for/of  information  (e.g.,  “Where  is 
the  enemy  aircraft”),  action  (e.g.,  “Take  the  bridge”),  or  coordination  (“I  need  your  help 
on  the  air  base”).  Raters  were  trained  prior  to  the  experiment  on  the  basis  of  tapes  and 
training  sessions  until  reliability  was  about  r  =  0.76. 

Workload.  At  five-minute  intervals  throughout  the  scenario,  each  team  member  was 
prompted  by  the  DDD  simulator  to  provide  an  estimate  of  the  workload  they  were 
experiencing  at  the  moment.  The  prompt  was  in  the  form  of  a  window  that  opened  on  the 
display.  The  workload  rating  was  made  on  a  seven  point  Likert  scale  anchored  at  on  end 
by  the  word  “low”  and  the  other  end  by  “high.”  The  windows,  which  were  mildly 
intrusive,  could  not  be  closed  until  a  rating  was  made. 

Participant  Impressions.  Following  the  final  session,  participants  filled  out  a  brief  survey 
asking  about  their  impressions  regarding  indications  of  reduced  performance  and  their 
impressions  regarding  potential  ways  to  improve  performance. 

Results 

Overall,  we  predicted  that  relative  to  the  congruent  cases  (D-d,  F-f),  performance  in  the 
incongruent  cases  (D-f,  F-d)  would  be  characterized  by  increased  communications  and 
perceived  levels  of  workload  due  to  the  model-based  coordination  requirements 
manipulation.  Due  to  these  increased  coordination  requirements,  we  also  expected  that 
performance  would  be  worse  in  the  incongruent  cases.  Accordingly,  due  to  our  interest  in 
congruence,  we  focused  our  analyses  on  the  interaction  between  organizational  structure 
and  scenario.  Below,  we  present  our  results  with  this  emphasis  in  mind.  In  general,  we 
evaluated  the  results  through  a  set  of  two-within-subjects  (scenario  and  replication)  by 
one-between-subjects  (structure)  mixed-model  multivariate  analyses  of  variance.  When 
computing  and  reporting  means  we  averaged  across  replications.  We  begin  by  evaluating 
the  primary  predictions,  and  then  address  communication  strategies  in  detail. 

Overall  Communications 

Based  on  the  modeling  approach,  we  predicted  that  the  overall  level  of  communications 
should  be  higher  in  the  mismatched  cases  because  coordination  requirements  were 
manipulated  to  be  much  higher  (i.e.,  players  would  have  to  communicate  verbally  to 
process  tasks  requiring  shared  resources).  Consistent  with  this  prediction,  averaged  over 
the  entire  scenario,  the  overall  volume  of  communications  was  higher  in  the  incongruent 
cases  than  in  the  congruent  cases,  as  demonstrated  by  a  significant  structure  by  scenario 
interaction,  F  (1,  6)  =  38.75,  p  <  0.002.  This  result  indicated  that  the  teams  talked  more 
when  faced  with  incongruent  situations  (Figure  2).  In  addition,  there  was  a  greater 
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volume  of  communications  for  the  d  scenario,  F  (1,6)  =  6.49,  p  <  0.05,  and  a  marginally 
significant  main  effect  for  structure,  F  (1,  6)  =  3.73,  p  <  0.1.  These  results  suggested  that 
there  were  more  communications  in  the  D  organization.  Note  in  particular  that  within  the 
F  organization,  the  volume  of  communications  almost  doubled  when  going  from  the  f  to 
the  d  scenario,  whereas  within  the  D  organization,  when  going  from  the  d  to  the  f  scenario 
communications  also  increased,  but  to  a  much  lesser  degree.  Thus,  given  these  scenarios, 
in  response  to  incongruence,  D  organizations  talked  more,  but  F  organizations  talked  a  lot 
more. 


□  d_scenario  n^scenario 


Organizational  Structure 


Figure  2.  Mean  communication  frequency  for  organizational  structure  and  scenario. 
Overall  Workload 

We  also  predicted  that  the  overall  level  of  perceived  workload  should  be  higher  in  the 
mismatched  cases:  Since  coordination  demands  were  manipulated  through  the  modeling 
process  to  be  high  in  the  mismatched  cases,  this  would  result  in  increased  communication 
requirements  and  an  increased  need  for  coordinated  actions  to  prosecute  mission  tasks. 
Hence,  we  predicted  that  the  perceived  workload  should  be  high  in  the  mismatched  cases. 
Consistent  with  this  prediction,  the  overall  level  of  perceived  workload  tended  to  be 
higher  in  the  mismatched  conditions,  although  this  result  was  only  marginally  significant, 
F  (1,  6)  =  2.59,  p  <  0.08.  As  shown  in  Figure  3,  teams  performing  with  the  D  structure 
tended  to  report  higher  workloads  when  performing  the  f  scenario  (incongruent 
condition)  as  compared  to  when  they  were  performing  the  d  scenario  (congruent 
condition),  while  teams  performing  with  the  F  structure  tended  to  report  higher  workload 
when  performing  the  d  scenario  (incongruent  condition)  as  compared  to  when  they 
performed  the  f  scenario  (congruent  condition).  There  were  no  significant  main  effects 
for  either  structure  (F  (1,6)  =  0.35,  ns)  or  scenario  (F  (1,6)  =  0.04,  ns).  Collectively, 
these  data  provide  some  limited  support  for  the  workload  hypothesis. 
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Figure  3.  Mean  perceived  workload  for  organizational  structure  and  scenario. 

Overall  Performance 


Overall,  we  predicted  that  performance  should  be  worse  in  the  mismatch  cases  because 
the  greater  requirements  for  coordination  should  lead  to  more  communication  and  higher 
workload,  resulting  in  some  decrements  in  the  ability  to  perform  multiple  tasks  quickly 
and  accurately.  Consistent  with  this  prediction,  we  found  that  the  overall  percentage  of 
tasks  completed  tended  to  be  superior  in  the  matched  conditions,  F  (1,  6)  =  50.37,  p  < 
0.001.  This  measure,  defined  as  the  number  of  tasks  attacked  divided  by  the  number  of 
tasks  arrived  in  the  simulation  (times  100),  indicated  that  in  both  organizations, 
performance  was  superior  in  the  congruent  cases  (Figure  4).  The  main  effect  for 
structure  was  not  significant  (F  (1,  6)  =  0.21,  ns),  but  there  was  a  marginally  significant 
main  effect  for  scenario  (F  (1,  6)  =  3.71  ,  P<0.  1).  Collectively,  these  data  suggest  that 
teams  were  less  successful  at  processing  the  tasks  that  were  present  in  the  incongruent 
cases. 


However,  with  respect  to  measures  of  mean  accuracy  for  tasks  attacked  (resources 
applied/resources  needed  for  tasks  processed)  and  mean  latency  for  tasks  attacked  (time 
of  task  processing  minus  time  of  task  arrival  in  the  game  for  tasks  processed),  there  were 
no  significant  main  effects  or  interactions,  although  with  respect  to  accuracy,  the  main 
effect  for  scenario  was  marginally  significant,  F  (1,  6)  =  5.70,  p  <  0.06.  Nevertheless, 
there  was  a  small  trend  toward  decreased  accuracy  for  organization  D  in  the  incongruent 
condition.  Overall,  these  data  collectively  suggest  that  there  was  a  tradeoff  between 
tasked  processed,  accuracy,  and  latency.  For  both  organizational  structures,  while  the 
percentage  of  tasks  processed  suffered  in  the  incongruent  conditions,  there  were  no 
significant  drops  in  accuracy  and  latency  for  the  tasks  that  were  actually  processed. 

In  total,  therefore,  these  data  indicated  that  the  congruence  manipulation  was  effective  in 
that  as  predicted,  performance  was  worse  in  the  incongruent  conditions  with  respect  to 
the  percentage  of  tasks  processed  (fewer  things  got  done).  Moreover,  as  predicted  by  the 
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model-based  manipulation  of  coordination  requirements,  communications  increased  in 
the  incongruent  cases,  especially  for  the  F  organization.  Similarly,  perceived  workload 
also  tended  to  increase  in  the  incongruent  conditions.  Given  these  findings,  which 
suggest  that  the  fundamental  model-based  manipulation  of  organizational  congruence 
was  effective,  we  now  turn  to  the  communication  strategies  the  organizations  exhibited 
given  the  scenario/structure  pairings. 


□  d_scenario  0f_scenario 
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Figure  4.  Mean  percentage  of  tasks  completed  for  structure  and  scenario. 


Communication  Strategies 

Within  the  context  of  this  investigation,  the  organizations  were  not  permitted  to  adapt 
their  structures.  Nevertheless,  the  organizations  were  free  to  adapt  their  strategies  as  they 
attempted  to  deal  with  the  mismatched  situations.  Consequently,  we  believe  that  these 
strategies  might  offer  insight  into  leading  indicators  because  they  reflect  how  an 
organization  attempts  to  cope  with  incongruence. 

To  explore  this  issue,  we  focused  our  analyses  in  particular  on  communication  patterns, 
for  communication  is  the  primary  mechanism  through  which  the  organizations  could 
express  their  attempts  to  deal  collectively  with  the  misfit  situation.  As  noted  above,  in 
response  to  incongruence,  both  the  D  and  F  organizations  talked  more  than  they  did  in 
congruent  situations.  However,  the  F  organizations  tended  to  talk  a  great  deal  more  in 
the  F-d  case  as  compared  to  the  F-f  case.  To  further  explore  this  issue,  we  now  address 
not  only  the  volume  of  communications,  but  also  the  patterns  of  communications. 

Communications:  Who  Talked  to  Whom?  To  explore  the  communication  patterns,  we 
plotted  the  communication  frequencies  for  each  pair  of  DMs  for  each  scenario-structure 
pairing.  Example  plots  are  shown  in  Figures  5  and  6.  We  then  conducted  six 
MANOVAs  on  the  various  pair  wise  groupings  (Green  to  others,  Purple  to  others,  Orange 
to  others,  Blue  to  others,  Red  to  others,  and  Brown  to  others).  To  clarify,  Green  was  the 
CVN  in  D  and  Strike  in  F;  Blue  was  DDGA  in  D  and  BMD  in  F;  Purple  was  DDGB  in  D 
and  1SR  in  F;  Red  was  CG  in  D  and  AWC  in  F;  Orange  was  FFG  in  D  and  SuWC/Mines 
in  F;  and  Brown  was  DDGC  in  D  and  SOF/SAR  in  F.  In  general,  these  analyses 


13 


indicated  that  for  the  various  positions,  communications  tended  to  significantly  increase 
with  one  or  more  other  positions  in  at  least  one  of  the  incongruent  conditions.  However, 
the  nature  of  the  increases  depended  on  the  scenario  and  structure  pairings.  For  example, 
although  Green  talked  to  Brown  a  bit  more  in  D-f  than  D-d  (there  were  no  differences  in 
Brown  talking  to  Green),  Brown  and  Green  talked  a  lot  more  to  each  other  in  F-d  as 
compared  to  F-f.  For  these  Brown-Green  pairings,  these  interactions  between  structure 
and  scenario  were  significant,  p  <  0.01.  Importantly,  the  model-based  design  process 
related  to  coordination  requirements  predicted  many  of  these  differences.  For  instance, 
by  definition,  Green  and  Brown  had  to  coordinate  more  in  F-d  than  F-f  based  on  resource 
requirements  that  demanded  cooperation.  Collectively,  these  data  suggest  that,  relatively 
speaking,  who  talked  to  who  in  response  to  incongruence  depended  on  organizational 
structure  and  the  particular  mission  scenario  demands. 


HD_d  □  D_f  □  F_d  □  F_f 


Figure  5.  Green’s  communications  with  other  team  members. 
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Figure  6.  Brown’s  communications  with  other  team  members. 
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Communications:  Talking  about  What?  To  explore  the  content  of  communications,  we 
used  survival  analysis  techniques  (Carroll,  1983).  In  brief,  by  looking  at  the  lengths  of 
time  between  each  player's  initiation  of  communications,  we  derived  DM-specific  as  well 
as  overall  condition-specific  waiting  time  distribution  functions  (Tuma  and  Hannan, 
1984),  which  in  turn  yielded  incidence  rate  estimates  for  communications.  Looking  at 
the  condition-specific  communication  rates,  congruent  and  incongruent  conditions  by 
architecture  were  statistically  compared  using  the  Mantel-Haenszel  test  (Dupont,  2002). 
This  analysis  once  again  showed  significant  differences  between  congruent  and 
incongruent  communication  rates  in  both  architectures,  controlling  for  DM-specific 
effects  (D:  X2(5)  =  32.40,  p  <  0.0001;  F:  X2(5)  =  206.95,  p  <  0.0001).  To  gain  insight 
into  general  communication  patterns  as  well  as  topic-specific  communication  patterns, 
we  analyzed  both  an  aggregation  of  all  communication  types,  and  separately  for  the 
coded  communication  categories.  We  looked  for  changes  in  a  given  player's  rate  of 
initiating  communications  relative  to  that  of  his  or  her  team  between  the  congruent  and 
incongruent  conditions.  First,  we  constructed  each  player's  role-relative  incidence  rate 
for  communication  as  the  ratio  of  an  individual  player's  communication  incidence  rate 
within  one  experimental  condition  to  the  team's  communication  incidence  rate  for  that 
condition.  Then,  we  looked  at  the  change  in  this  ratio  from  the  congruent  condition  to  the 
incongruent  condition.  Figures  7  and  8  illustrate  large  ratio  changes  (absolute  value  > 
50%)  for  the  Divisional  and  Functional  architectures,  respectively  (the  larger  the  increase 
or  decrease,  the  more  +’s  or  -’s,  respectively). 
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Figure  7.  Role-relative  incidence  rates  for  incongruence  /role-relative  incidence  rates  for 
congruence  for  Organization  D.  This  ratio  indicates  the  change  in  role-relative 
probability  from  congruent  to  incongruent  scenarios. 

In  general,  this  analysis  revealed  a  strong  distinction  in  the  nature  of  adaptation  to  the 
incongruent  scenario.  For  the  D  structure,  the  role-relative  communication  rates  by  DM 
for  communication  type  did  not  change  drastically  between  the  congruence  and 
incongruent  scenarios.  Of  the  30  observed  changes  in  relative  incidence  rates  (6  DMs  by 
5  communication  types),  there  were  large  changes  for  only  two  conditions  (Figure  7). 
The  columns  in  Figure  7  refer  to  all  communications,  communications  about  enemy 
tasks,  communications  about  friendly  assets,  requests  for  information,  and  non-requested 
transfers  of  information.  In  contrast,  for  the  F  structure,  there  were  strong  changes  in 
communication  types  between  the  congruent  and  incongruent  scenarios  for  many 
categories.  Of  the  30  observed  changes  in  relative  incidence  rates,  there  were  large 
changes  for  21  conditions  (Figure  8).  These  data  indicated  that  when  F  organizations 
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went  from  f  to  d,  the  content  of  communications  changed  substantially.  However,  when 
D  organizations  went  from  d  to  f,  communications  content  changed  in  more  subtle  ways. 
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Figure  8.  Role-relative  incidence  rates  for  incongruence  /role-relative  incidence  rates  for 
congruence  for  Organization  F.  This  ratio  indicates  the  change  in  role-relative 
probability  from  congruent  to  incongruent  scenarios. 

Collectively,  these  analyses  of  the  communications  data  showed  that  when  D 
organizations  went  from  congruent  to  incongruent  situations,  they  reacted  by  talking 
more.  However,  the  changes  in  the  likelihood  of  particular  pairs  of  DMs  talking  to  each 
other  were  relatively  small  as  were  the  changes  in  communications  content.  In  contrast, 
when  F  organizations  went  from  congruent  to  incongruent  situations,  they  reacted  by 
talking  more,  making  substantial  changes  in  whom  they  talked  to,  and  making  substantial 
changes  in  what  they  talked  about.  These  data  suggest  that  the  context  of  being  in  a 
particular  organization  given  a  particular  mission  influences  how  organizations  react 
when  trying  to  cope  with  incongruence. 

Participant  Impressions 

To  further  assess  the  manipulation,  and  potential  reactions  to  incongruence,  a  follow-up 
questionnaire  was  administered  to  participants  at  the  end  of  the  experiment  to  obtain 
information  regarding  what  cues  participants  were  aware  of  that  specified  that 
performance  was  beginning  to  degrade.  In  addition,  participants  were  also  asked  what 
type  of  changes  they  thought  could  have  improved  performance  at  the  point  in  the 
scenario  when  they  noticed  that  performance  was  beginning  to  degrade.  This  information 
speaks  to  potential  leading  indicators  of  the  need  to  change. 

More  specifically,  with  respect  to  impressions  regarding  indicators  of  performance 
decrement,  we  collected  seventy  responses  from  48  participants,  which  were  tallied  and 
categorized  into  the  categories  shown  in  Table  2.  Two  major  categories  of  response, 
uncompleted  tasks/less  than  optimal  performance  (51%)  and  communication/ 
coordination  demands  (23%),  comprised  74  %  of  the  responses.  Examples  of  the  types  of 
responses  listed  under  uncompleted  tasks/less  than  optimal  performance  included  items 
such  as  loss  of  situational  awareness,  not  prosecuting  tasks,  own  platforms  being  hit  by 
the  enemy,  being  in  a  reactive  mode  (i.e.,  behind  the  timeline),  not  accomplishing 
unanticipated  and  pop-up  missions,  not  detecting  SCUD  launches  in  time  to  engage  them, 
lapse  of  time  for  use  of  an  asset,  failure  to  complete  time-critical  tasks,  and  not  engaging 
a  task  simultaneously  (as  required).  With  respect  to  communications/coordination 
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demands,  responses  included  items  such  as  increased  communications,  too  much  chatter, 
loss  of  communications  discipline  (e.g.,  people  talking  over  each  other,  not  using  call 
signs),  talking  faster/louder/  increased  inflection,  slow  radio  response  from  others,  and 
the  inability  to  coordinate  due  to  increased  communications.  Collectively,  these 
responses  were  consistent  with  the  findings  reported  above,  and  in  addition,  they  point  to 
performance  decrements  and  communications  as  being  likely  sources  of  leading 
indicators. 


Indicators  of  Degraded  Performance 

Frequency 

Uncompleted  tasks/  less  than  optimal  performance 

34 

Communication  and  Coordination  demands 

16 

Workload 

6 

Process/  structure 

2 

Training 

2 

Other 

7 

Table  2.  Participant  impressions  regarding  indicators  of  degraded  performance. 


Suggestions  to  Improve  Degraded  Performance 

Frequency 

Adapt  Process  or  Structure 

23 

Provide  Decision  Aids  and  Human-Computer  Interaction  Improvements 

16 

Communications  Changes 

12 

Change  Asset  Allocation 

7 

Training 

1 

Other 

12 

Table  3.  Participant  suggestions  to  improve  degraded  performance. 

When  asked  about  what  type  of  changes  participants  thought  could  have  improved 
performance  at  the  point  in  the  scenario  when  they  noticed  that  performance  was 
beginning  to  degrade,  participants  once  again  made  a  variety  of  responses  that  we 
grouped  into  the  categories  shown  in  Table  3.  One  third  (thirty-two  percent)  of  the 
suggestions  recommended  by  participants  to  improve  degraded  performance  pertained  to 
adaptation  of  either  the  process  or  the  organizational  structure,  consistent  with  the  goals 
of  the  research  program.  Along  these  lines,  several  suggestions  involved  allowing  the 
burden  of  tasks  to  be  shifted  to  less-tasked  people  —  on  a  task-by-task  basis  —  so  that 
when  one  individual  became  overloaded  they  could  shift  assets  to  another  individual  who 
would  then  perform  the  task  in  order  to  level  the  workload.  Several  participants  (who 
performed  in  the  D  structure)  recommended  that  coordinator  positions  be  established  for 
warfare  areas  along  functional  vice  geographic  lines.  In  addition,  twenty-three  percent  of 
the  suggestions  regarding  ways  to  improve  degraded  performance  involved  providing 
decision  aid  support  to  provide  an  automated  means  to  inform  the  participant  regarding 
various  aspects  of  the  scenario.  These  suggestions  included  informing  the  participants 
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regarding  the  following  conditions:  (1)  when  assets  were  available  for  each  mission 
(assets  available  at  that  time  in  the  scenario),  (2)  that  an  asset  is  out  of  ammunition 
(immediately  after  it  conducts  its  last  engagement),  (3)  that  tasks  are  waiting  to  be 
completed,  and  (4)  that  the  time  remaining  to  complete  a  task  is  low  (e.g.,  five  minutes 
remaining). 


Discussion 

Based  on  the  modeling  work  that  lead  to  the  organizational  definitions  and  the  scenarios, 
we  predicted  that  relative  to  the  matched  cases  (D-d  and  F-f),  performance  in  the 
mismatched  conditions  (D-f  and  F-d)  would  be  characterized  by  increased 
communications  and  increased  perceived  workload.  In  addition,  we  predicted  that 
performance  would  be  worse  in  the  mismatched  conditions  due  to  these  organization- 
scenario  demands  and,  more  generally,  the  limitations  imposed  by  the  lack  of  fit  between 
the  organizational  structures  and  the  mission  scenarios. 

Overall,  our  results  supported  these  predictions.  Communications  were  increased  in  the 
mismatched  conditions,  and  consistent  with  these  findings,  the  participants  remarked  on 
the  communications  problems  associated  with  the  incongruent  conditions.  Importantly, 
this  change  in  communications  volume  likely  reflected  the  model-based  manipulation  of 
the  need  for  cross-participant  coordination,  which  was  the  primary  tool  employed  to 
engineer  congruence.  Similarly,  consistent  with  these  manipulations,  the  perceived 
workload  also  increased,  although  this  trend  was  only  marginally  significant.  Once  again, 
these  data  were  generally  consistent  with  the  model-based  manipulation  of  coordination 
requirements.  With  respect  to  measures  of  performance,  which  included  the  percent  of 
tasks  processed,  latencies,  and  accuracy,  results  showed  reduced  performance  in  the 
mismatched  conditions.  Participants  in  both  organizations  processed  fewer  tasks  in  the 
incongruent  cases,  and  once  again,  participants  in  both  organizations  remarked  on  the 
performance  decrements  associated  with  the  incongruent  scenarios. 

In  total,  these  findings  were  consistent  with  the  model-based  predictions.  Given  this 
global  pattern  of  results,  we  take  these  data  as  strong  evidence  that  the  congruence 
manipulation  was  effective.  Moreover,  the  validation  of  these  predictions  now  provides 
us  with  our  “license”  to  further  explore  in  detail  how  the  organizations  studied  attempted 
to  cope  with  incongruence,  thus  shedding  light  on  potential  “leading  indicators”  of  the 
need  for  organizational  adaptation.  Here,  we  address  measures  averaged  over  the  entire 
mission  scenarios.  In  our  companion  paper,  Entin  et  al.  (2003),  we  address  measures 
taken  over  time. 

Strategies  as  Leading  Indicators 

Within  the  context  of  the  present  investigation,  although  the  organizations  were  not 
permitted  to  adapt  their  structures,  they  were  nevertheless  free  to  adapt  their  strategies  as 
they  attempted  to  deal  with  the  mismatched  situations.  We  believe  that  these  strategies 
might  offer  insight  into  leading  indicators  of  the  need  for  change  to  the  extent  that  they 
reliably  reflect  how  an  organization  attempts  to  cope  with  incongruence.  In  general,  we 
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define  leading  indicators  as  measures  that  signal  that  an  organization  is  “out  of 
alignment”  with  its  mission  and  needs  to  change  its  structure  to  improve  performance  and 
optimize  mission  effectiveness.  Note  that  these  indicators  may  be  “leading”  or  “lagging” 
to  various  degrees.  Ideally,  leading  indicators  would  predict  the  need  for  change  prior  to 
any  performance  decrement.  However,  some  indicators  may  first  surface  as  performance 
begins  to  decline.  For  instance,  the  performance  decrements  that  the  participants 
remarked  on  were  likely  in  this  category.  Ultimately,  measures  taken  over  time  will  help 
to  clarify  the  extent  to  which  particular  items  are  truly  leading  or  lagging  (Entin  et  al., 
2003). 

In  this  paper,  to  explore  the  issue  of  leading  indicators,  we  focused  our  work  on 
communication  strategies.  By  strategy,  we  simply  mean  the  responses  that  the 
organizations  made  in  their  communications  as  they  coped  with  incongruence.  We 
focused  on  communications  because  they  are  the  primary  mechanisms  through  which  the 
organizations  could  express  their  attempts  to  deal  collectively  with  the  misfit  situations. 
Our  results  showed  that  both  organizations  talked  more  when  faced  with  incongruence. 
While  the  D  organizations  talked  more,  however,  the  F  organizations  talked  a  great  deal 
more,  there  were  strong  differences  in  the  extent  to  which  different  people  talked  to  each 
other,  and  there  were  robust  changes  in  the  content  of  the  communications.  Thus,  in 
trying  to  cope  with  the  incongruence,  the  F  organization  reacted  differently  when  going 
from  f  to  d  than  the  D  organization  when  going  from  d  to  f.  The  organizations  reacted 
by  increasing  communications,  but  the  specific  contexts  of  the  missions  and 
organizational  structures  influenced  the  nature  of  those  reactions. 

This  finding  -  that  the  context  mattered  -  is  critically  important,  for  it  suggests  that  the 
leading  indicators,  or  symptoms  of  incongruence,  may  be  complex  in  that  they  will  be 
shaped  by  the  interaction  of  mission  tasks  and  organizational  structures,  as  well  as  a 
number  of  other  potential  factors  such  as  history  of  the  organization,  individual 
proficiencies,  fatigue,  etc.  This  means  that  the  appearance  of  a  particular  leading 
indicator  such  as  communication  content  may  be  probabilistic  rather  than  absolute,  such 
that  it  might  take  a  set  of  measures  rather  than  any  one  in  particular  to  ascertain  the  state 
of  congruence.  We  expect  that  this  will  be  particularly  true  when  larger  and  more 
complex  organizations  face  larger  and  more  complex  missions  in  the  “real  world.” 
Consequently,  it  is  imperative  to  ascertain  how  organizational  structures  and  mission 
scenarios,  which  may  be  more  or  less  congruent  or  incongruent,  interact  to  shape  the 
behaviors  observed,  for  these  behaviors  (e.g.,  strategies)  will  only  serve  as  valid  leading 
indicators  to  the  extent  that  we  understand  how  contextual  factors  shape  organizational 
performance  and  process. 

Next  Steps 

Building  on  these  data,  the  mission  scenarios,  and  the  organizations,  we  believe  that  we 
are  now  well  positioned  to  effectively  study  structural  adaptation.  In  creating  the 
scenarios  examined  here,  our  hope  was  to  engineer  conditions  that  made  it  advantageous 
for  an  organization  to  adapt  structure.  Our  reasoning  was  that  when  the  structure  of  an 
organization  becomes  too  incongruent  with  its  mission,  an  organization  should  “feel  the 
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pain”  and  be  motivated  to  change  to  enhance  mission  effectiveness.  Collectively,  our 
results  suggest  that  we  have  indeed  created  conditions  in  which  organizations  are 
burdened  by  coordination  requirements.  These  results  suggest  that  if  given  a  chance  to 
adapt  when  faced  with  misalignment  (D-f,  F-d),  the  organizations  studied  here  might  be 
sufficiently  motivated  to  change  their  organizational  structures  in  order  to  gain  a  state  of 
congruence  (F-f,  D-d). 

However,  while  there  is  evidence  that  organizations  will  institute  strategy  changes,  such 
as  those  noted  above,  they  also  appear  to  be  quite  reluctant  to  make  changes  to  their 
structures  (Entin,  Serfaty,  and  Kerrigan,  1998;  Hollenbeck  et  al.,  1999).  Hence,  either 
adaptation  needs  to  be  imposed  on  an  organization  from  above,  or  alternatively,  it  is 
necessary  to  provide  substantial  support  that  moves  the  organization  in  the  direction  of 
change.  Thus,  in  order  to  successfully  induce  and  study  organizational  adaptation  that  is 
not  imposed  on  an  organization,  it  is  necessary  to  not  only  create  a  state  of  organizational 
pain,  but  also  direct  the  organization’s  awareness  to  the  source  of  that  pain,  provide  them 
with  the  knowledge  that  another  mode  of  organization  would  be  superior,  and  provide 
them  with  the  tools  to  adapt.  Similarly,  in  the  case  of  change  imposed  on  an  organization 
from  the  outside  (e.g.,  via  a  chain  of  command),  it  is  necessary  to  understand  the 
behaviors  of  organizations  coping  with  incongruence  such  that  a  commander  could  make 
a  decision  to  impose  change  based  on  an  understanding  of  what  incongruence  looks  like 
in  action.  Accordingly,  building  on  these  data  and  those  presented  in  Entin  et  al.  (2003), 
we  are  now  exploring  both  a  variety  of  measures  that  could  serve  collectively  as  potential 
leading  indicators  and  different  ways  of  presenting  that  information  to  organizations  to 
support  structural  adaptation.  In  future  empirical  work  our  goal  will  be  to  study  the 
processes  of  on-line  structural  adaptation  by  directing  attention  toward  critical  leading 
indicators  of  the  need  to  change  and  by  exploring  how  organizations  need  to  be  prepared 
in  order  to  change  effectively. 

In  summary,  this  set  of  analyses  indicates  that  our  model-based  manipulations  of 
organizational  congruence  were  successful.  Our  results  showed  that,  as  predicted, 
relative  to  the  congruent  cases,  the  incongruent  cases  indeed  lead  to  less  effective  mission 
performance.  These  results  demonstrated  the  power  of  model-based  organizational 
design  for  optimization  of  mission  effectiveness.  Moreover,  based  on  this  successful 
manipulation,  this  work  sets  the  stage  for  further  work  on  structural  adaptation.  Our  goal 
is  to  study  the  processes  that  underlie  adaptation  when  an  organization  moves  from  one 
structure  to  another.  The  work  reported  here  provides  the  organizational  structures  and 
scenarios  necessary  to  explore  adaptation,  and  importantly,  the  data  on  which  to  further 
study  what  organizational  congruence  looks  like  in  action. 
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Questions  &  Objectives 


1  Do  model-based  predictions  of  (in)  congruence 
produce  measurable  difference  in  process  and 
outcome? 


Organization 


Mission 


■  Measure  the  effects  of  congruence  on 
organizational  performance  and  processes 

■  Lay  the  foundation  for  further  work  on  adaptation 

■  Establish  the  conditions  for  change 

■  Identify  leading  indicators  of  incongruence 

■  How  do  we  support/induce  adaptation? 


Divisional  (D) 


Organizational  Structures 
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Functional  (F) 


1 

2 

3 

4 

5 

6 

Platform 

STRIKE 

ISR 

AWC 

SuWC/MINES 

SOF/SAR 

1 

CVN 

2F18S 

XXX 

1UAV 

2F18A,  E2C 

1FAB,  1MH53 

1HH60 

2 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1HH60,1SOF 

3 

DDGB 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1HH60,1SOF 

4 

CG 

8TLAM 

3ABM 

1UAV 

6SM2 

1  FAB,2HARP,1  MH53 

1HH60 

5 

2F18S 

XXX 

1UAV 

2F18A,E2C,4SM2 

1  FAB,2HARP,1  MH53 

1HH60 

6 

DDGC 

8TLAM 

3ABM,4TTOM 

1UAV 

6SM2 

1FAB,  2HARP 

1HH60,1SOF 

Asset  “ownership”  and  control  shape  team  structure: 

*  Multi-function  vs.  single-function  responsibilities 
■  Geographic  Area  of  Responsibility:  Local  vs.  Global 

■  Heterarchical,  not  Hierarchical,  organization 


Scenarios 
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■  Congruence  Manipulation 

■  Capitalizes  on  Roles  and  Geography  (task  and  asset 
locations) 


i 


Strategies  for  Manipulation 

■  Coordination  Requirements 

■  Task  Phasing 

■  Boundary  Splitting 
*  Limited  Assets 


of  Congruence 


1  w  Is? 
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-L  Itumlh 
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Defend 
own  assets 


Functional  (f)  Scenario 


Obstacles 

2SOF 

to  SOF 

. 

j  •  RGF 

f  BLOw\ 

““ ^Ibridge/ 

L . 

=  aggregated  defend  task,  showing  possible  subtasks 
o  =  aggregated  encounters  task,  with  possible  subtasks 
o  =  mission  tasks  (that  must  be  done);  known  in  advance 
0  =  GEVA  may  spawn  as  a  result  of  performing  task 


2SOF+ 

2FAB 


3SOF  2SOF+2FAB 

*  indicates  that  these  must  be  distinguished  from  neutral  (or  decoy)  counterparts 


TASK  RESOURCE  REQMTS 

SPG:  2  ASuW 
SPT,  SPH:  1  ASuW 
SGUN:  2  FAB 
SSAR:  2SAR 
SMIN:  2  MINES 

GEVA :  2SAR 

GCDL,  GSML:  1  STRK 

GSAM:  2  TLAM  (from  2  different  platforms) 

GSA3:  2  STRK  (1  F18S) 

GSA6:  2  TLAM  (from  2  different  platforms) 
GRGF\  3  STRK 

AAC,  APH,  ACDM,  AXOC:  1  AAW 
ACAP:  3AAW 
AMIS:  1  ABM 

-  other  unanticipated  tasks  via  HELP 
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Divisional  (d)  Scenario 


Defend 
own  assets 


Obstacles 
to  SOF 


1SOF+2STRK 


1SOF+2STRK  +1FAB 


=  aggregated  defend  task,  showing  possible  subtasks 
o  =  aggregated  encounters  task,  with  possible  subtasks 
o  =  mission  tasks  (that  must  be  done);  known  in  advance 
GEVA  may  spawn  as  a  result  of  performing  task 


TASK  RESOURCE  REQMTS 

SDG:  IASuW+IAAW 
SPT,  SPH:  1  ASuW 
SHOS:  1  SAR  +  1  FAB 
SSAR:  1  SAR  +  1  FAB 
SMIN:  1  MINES  +±F18A 

GEVA:  1  SAR  +  1_  F18A 
GCDL,  GSML:  1  STRK 
GSAM:  1  TLAM  +  1  SOF 
GSA3:  2  STRK  (1  F18S) 

GSA6:  2  TLAM  (from  2  different  platforms) 
GRGF\  2  STRK 

AAC,  APH,  ACDM,  AXOC:  1  AAW 
ACAP:  2AAW 
AMIS:  1  ABM 

-  other/unanticipated  tasks  via  HELP 


1SOF+ 

2STRK+ 

1FAB 


indicates  that  these  must  be  distinguished  from  neutral  (or  decoy)  counterparts 
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Experimental  Design 


>  48  participants  organized  into  eight  6- 
person  teams 

■  Independent  Variables 

»  Structure  is  a  between  subjects  variable 

■  Divisional  (D)  vs.  Functional  (F) 

•  Scenario  is  a  within  subjects  variable 

■  Divisional  favoring  (d)  vs.  Functional  favoring  (f) 

->  Congruence  is  the  interaction  of  structure  and 
scenario. 


Experimental  Design 
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■  Procedure 

■  “Buttonology”  (2  hours) 

■  “Hash”  (2  hours) 

1  First  Replication  (2  hours) 

1  Second  Replication  (2  hours) 

■  Design 

Scenario 
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Results 


■  Analyses  focused  on  communications,  workload, 
and  performance. 

■  Emphasis  on  model-based  predictions 
»  Emphasis  on  patterns 

■  Emphasis  on  communications  because  they  directly 
reflect  strategy  adaptations,  and  are  thus  a  strong 
candidate  for  leading  indicators 

1  Overall,  results  showed  that  in  the  incongruent 
cases,  communications  increased,  workload 
increased,  and  performance  worsened. 

■  However,  the  context  mattered. 

■  Structure/scenario  pairings  influenced  reaction  to 
incongruence. 


Overall  Performance 
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■  As  predicted  based  on  the  model  design  process, 
performance  was  worse  in  the  incongruent  cases. 


□  d  scenario  af  scenario 
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co 


CD 
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CD 
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Organizational  Structure 
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Communications  (Talking  More?) 


I 


The  manipulation  of  congruence  hinged  on 
coordination  requirements. 

*  Model-Based  Prediction:  Since  more  coordination  required 
in  incongruent  cases,  there  should  be  more  communication 
in  incongruent  cases. 


□  d  scenario  nf  scenario 


Organizational  Structure 


->  Bigger  change  in  Functional 
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Communication  Patterns 


■  Organizational  Structure  Mattered 

■  In  response  to  incongruence: 

■  Divisional  teams  talked  more. 

■  However,  Functional  teams  talked  a  lot  more,  and 
there  were  stronger  differences  in  who  talked  to 
who  about  what. 


The  context  of  structure  and  scenario  mattered. 


Average  Communication 
Frequency 
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Communications:  Talking  About  What? 

•  Incidence  Rate  Analysis  of  Communication  Patterns 

•  For  Divisional,  the  “role-relative”  incidence  rates  for  communications  by 
player  and  type  did  not  change  drastically  between  the  congruent  and 
incongruent  conditions.  Divjsiona|  rir_i/rir_c  Ratio 


(Indicates  change  in  role-relative  communication  probability  from  congruent  to 

incongruent  scenarios) 


DM 

All  Task  Asset  Req  Xfer 

Green 

Blue 

Purple 

Red 

Orange 

Brown 

•  For  Functional,  there  were  strong  changes  in  communication  types  for 
many  players  ->  Talking  about  different  things... 


Functional  RIR-I/RIR-C  Ratio 


DM 
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Communications:  Talking  About  What? 


•  Communication  &  Cooperation  Networks  illustrate  interaction  patterns 
within  the  team 

•  Player-Player  networks  are  created  when: 

•  A  Player  sends  a  communication  to  another  Player 

task 

•  For  the  Divisional  structure  the 
association  is  about  the  same  in 
congruent  and  incongruent 
conditions 

•  For  the  Functional  structure  the 
association  is  quite  different 
between  congruency  conditions 
and  largest  in  the  incongruent 
condition 


Players  cooperate  to  process  a 
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□  Congruent  □  Incongruent 


Divisional  Functional 

Organizational  Structure 


•  In  Divisional  there  is  little  evidence  for  strategy  adaptation.  However,  in 
Functional  there  is  a  radical  change  in  the  association  of  communication 
and  cooperation 


Perceived  Workload 
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•  The  manipulation  of  congruence  hinged  on  coordination 
requirements. 

Model-Based  Prediction:  Since  more  coordination  in  incongruent  cases, 
there  should  be  a  higher  perceived  workload  in  incongruent  cases. 


□  d  scenario  af  scenario 


p  =  0.08 


Organizational  Structure 
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Performance 


■  The  manipulations  of  congruence  were  successful 
in  changing  communications  and  perceived 
workload. 

•  These  changes  were  predicted  by  the  model-based 
manipulation  of  coordination  requirements. 

■  Given  these  changes  in  response  to  coordination 
needs,  we  expected  performance  to  be  worse  in 
the  incongruent  conditions. 

■  Will  performance  be  different  across  the  structure  and 
scenario  pairings? 
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Performance 


■  When  coping  with  incongruence,  performance  changes 
were  dependent  on  the  structure  &  scenario  pairings. 

■  Percent  of  attacks  processed  by  latency  for  tasks  processed 

■  Changes  in  tasks  processed  for  each  structure 

■  No  changes  in  latency  for  either  structure 
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■  Percent  of  tasks  processed  by  accuracy  for  tasks 
processed 

■  Changes  in  tasks  processed  for  each  structure 
•  Changes  in  accuracy  for  Divisional  only 
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■  Accuracy  by  latency  for  tasks  processed 

■  No  changes  in  latency 

■  Changes  in  accuracy  for  Divisional  only 
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Conclusions 


■  The  organizations  and  scenarios  studied  here 
set  the  stage  for  further  work  on  structural 
adaptation. 

■  Based  on  modeling  work  we  successfully  created  the 
conditions  under  which  change  is  needed. 

*  The  strategic  adaptations  to  incongruence 
depended  on  the  organizational  structure  & 
scenario  pairings. 

■  The  leading  indicators  may  be  complex  and  context 
dependent,  especially  when  larger  and  more  complex 
command  and  control  organizations  are  considered. 


